Repository navigation
Make checkpoint and delete survive control-plane and object-store disruption during upgrades - #1
Merged
Merged
Conversation
…telet forever - DeleteActor's ensureAteletTerminated dialed the actor's atelet on every reap, and when the node no longer existed the dial failed with Unavailable each time, so the reaper looped on a DELETING actor for good (seen on a customer's EKS cluster after a worker node pool was replaced during an upgrade) - The control-plane atelet dialer now consults a node lister: a dial to an atelet on a node the cluster has lost reports ErrNodeGone, and the delete workflow skips the terminate, since nothing on that node is left to terminate - ate-api-server's ClusterRole gains get, watch and list on nodes for the lister - TestDeleteActor_SkipsTerminateOnGoneNode covers the delete; the dialer test covers the node check Amp-Thread-ID: https://ampcode.com/threads/T-01a178d8-73f0-751b-8cb4-c5a11dd7ef41 Co-authored-by: Hitesh Sagtani <hitesh@ampcode.com>
…lable - The object-store plugin reported every storage SDK error whose retries were exhausted as Internal, so a GCS 503 during a checkpoint upload crashed the actor: the control plane retries only Unavailable, Canceled and DeadlineExceeded from atelet - HTTP 408, 429 and 5xx from the storage service now map to Unavailable; 4xx such as 403 stay Internal - Seen during the 2026-10-26 production Substrate upgrade, where five sandboxes were lost to a 503 burst on upload Amp-Thread-ID: https://ampcode.com/threads/T-01a178d8-73f0-751b-8cb4-c5a11dd7ef41 Co-authored-by: Hitesh Sagtani <hitesh@ampcode.com>
- A Checkpoint under UPLOAD_MODE_IMMEDIATE drove worker's CheckpointWorkload, which deletes the workload, and only then uploaded; when the upload failed, or the control plane lost the response, the retried Checkpoint drove worker again against a workload that was gone and the actor crashed (agent-substrate#372 anticipated this) - The checkpoint now moves worker's snapshot to the node's local snapshot directory and records it as LatestLocal before any upload, in both upload modes; a retried Checkpoint of a snapshot this node already committed skips worker and uploads the local copy, or returns the recorded state when it is durable already - The upload from the local copy goes through the promote path, so an object-store outage reaches the control plane as Unavailable and is retried rather than crashing the actor; a checkpoint that cannot keep a local copy still uploads from the checkpoint directory as before - BeginUploadingCheckpoint holds the actor's background uploads off while an immediate checkpoint uploads, so the same snapshot is never uploaded twice at once - TestSuspendUploadFailureRetriesFromLocalCopy covers a 503 on the first upload and the retry; TestFailedCheckpointKeepsLocalSnapshot now covers the move failing Amp-Thread-ID: https://ampcode.com/threads/T-01a178d8-73f0-751b-8cb4-c5a11dd7ef41 Co-authored-by: Hitesh Sagtani <hitesh@ampcode.com>
- ate-api-server drained for 15 s on SIGTERM, then force-stopped in-flight RPCs; a SuspendActor driving an atelet Checkpoint that streams a multi-GiB snapshot was cut mid-upload during the 2026-10-26 rollout, and the client's retry sent a Checkpoint for a workload worker had already deleted - --drain-timeout is now 5m, atelet's, with terminationGracePeriodSeconds 330 to cover --drain-delay plus the force stop - A Deployment rollout counts a terminating pod as gone, so the longer drain does not hold up the rollout Amp-Thread-ID: https://ampcode.com/threads/T-01a178d8-73f0-751b-8cb4-c5a11dd7ef41 Co-authored-by: Hitesh Sagtani <hitesh@ampcode.com>
A process past "Shutdown complete" flushed traces and metrics with a background context, so an unreachable collector held the pod until the kubelet's grace period. Give each provider 15s. Amp-Thread-ID: https://ampcode.com/threads/T-01a178d8-73f0-751b-8cb4-c5a11dd7ef41 Co-authored-by: Hitesh Sagtani <hitesh@ampcode.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four fixes from the 2026-10-26 production Substrate upgrade (amp run 40372718435 lost 5 sandboxes after f33168e fixed the first 6) and a customer's EKS upgrade:
Opened to run
run-testson a kernel with SEEK_DATA; the orb that wrote this cannot run the upload tests. Merges by fast-forward to main.